Externalizing redis to ssl enabled ElastiCache


#1

I’m following instructions at https://www.spinnaker.io/setup/productionize/caching/externalize-redis/ to replace in-cluster Redis cache with AWS ElastiCache & in-transit encryption enabled (spinnaker v1.8.1).
So far, all attempts (overrideBaseUrl on profiles/redis.yml, service.redis.host in service-settings/spinnaker-local.yml, using rediss instead of redis) to read from elasticache fail with connection reset and stack trace below.

I was able to attach a debugger to gate, step through code and identify that JedisConnectionFactory.useSsl is always set to false regardless of which way i configure it.

I’m wondering if anybody is using ElastiCache with ssl enabled, and what would be the steps to configure spinnaker components

Thanks,
Z

partial gate stacktrace:
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Connection reset
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:202) ~[jedis-2.9.0.jar:na]
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.BinaryJedis.auth(BinaryJedis.java:2139) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:108) ~[jedis-2.9.0.jar:na]
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:888) ~[commons-pool2-2.4.3.jar:2.4.3]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:432) ~[commons-pool2-2.4.3.jar:2.4.3]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361) ~[commons-pool2-2.4.3.jar:2.4.3]
at redis.clients.util.Pool.getResource(Pool.java:49) ~[jedis-2.9.0.jar:na]
… 139 common frames omitted
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[na:1.8.0_171]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_171]
at java.net.SocketInputStream.read(SocketInputStream.java:127) ~[na:1.8.0_171]
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:196) ~[jedis-2.9.0.jar:na]
… 150 common frames omitted


#2

Going out on a limb here, but can you over ride those with JVM parameters?


#3

Hi @zsisic,

Sorry you’re having troubles. I was unaware that AWS had it’s own managed Redis solution (thanks for the pointer!)

It appears that there isn’t a common Redis Configuration class across all of the services, meaning there also isn’t a single place to hook into the JedisConnectionFactory's useSsl field.

It does appear that @cfieber added a one-off redis.connection.secure configuration property in Gate. Cam, are you guys connecting to Redis over SSL? If so, how do you configure it?


#4

Yes, you should use the redis.connection.secure property in order to use Elasticache with TLS. This is something mentioned in several Spinnaker blog posts–including this deck from Stitch Fix: https://www.slideshare.net/DianaTkachenko/making-spinnaker-go-stitch-fix-67096311


#5

I already have the redis.connection.secure property set in my gate override file, but this would only let me use a single node non-ssl version of elasticache. I was able to trace the property back to bypassing the notify-keyspace-events configuration and hence the requirement to set it manually.
As @ttomsu points out, JedisConnectionFactory is programmatically instantiated in GateConfig without providing options to override useSsl flag, so i am not quite sure to enable this behavior as we can not rely on property override capabilities of spring boot.
Another interesting fact is that if deploying a redis version suggested by stitch-fix, it would not let us enable in transit ssl and password as those options are enabled for versions 3.2.6 and 4.x (which we’re trying to use)


#6

Your best bet may to file a feature request and either wait for someone in the community to pick it up or implement it yourself. The kork-jedis would be my first place to put common config code like this, though I suggest reaching out to the #dev channel in the chat room for further guidance.


#7

Hello again,

Just to clarify my goal, we wanted to back up the pipeline execution logs and ensure they can be recovered. Going back to the drawing board, i realized that i only need to update orca’s configuration to make this work, and have everything else (cached data) go through the internal redis instance as its state gets constantly rebuilt.

Good news is that kork-jedis already detects ssl based on proto and lib version is already pulled in for Spinnaker v1.8.1.

On top of that, Orca’s legacy redis configuration beans1 are marked as deprecated, and I traced the change back to PR with configuration necessary to switch the redis connection for execution repository to the external ssl enabled and password protected instance.
default/profiles/orca-local.yml

redis:
  clients:
    executionRepository:
      primary:
        driver: redis
        config:
          connection: rediss://:token@my-redis-url:6379

Thanks a lot for all the pointers,

Z