QuickGuide: Amazon Kinesis Setup

Previous guide in series – Amazon CloudWatch Setup

Step 1 – Download and extract Oracle JDK

Download Link -> Download Linux x64 Compressed Archive

Alternatively, you can download from here – https://demos3gl.s3.us-east-2.amazonaws.com/jdk-8u281-linux-x64.tar.gz

Use WinSCP or a similar tool to transfer to the Ubuntu EC2 instance (/opt)

cd /opt
ls -al
tar -zxvf jdk-8u281-linux-x64.tar.gz

Grab directory – /opt/jdk1.8.0_281

Step 2 – Download and extract Kinesis Agent

Download Link -> Click on Code -> Download zip

Alternatively, you can download from here – https://demos3gl.s3.us-east-2.amazonaws.com/amazon-kinesis-agent-master.zip

Use WinSCP or a similar tool to transfer to the Ubuntu EC2 instance (/opt/software)

cd /opt/software
sudo apt install zip -y
unzip amazon-kinesis-agent-master.zip
ls –al
cd amazon-kinesis-agent-master

Step 3 – Switch user to root and install Kinesis Agent

sudo su
export JAVA_HOME=/opt/jdk1.8.0_281
export PATH=/opt/jdk1.8.0_281/bin:$PATH
java -version
ls -al
./setup --install

Config file: /etc/aws-kinesis/agent.json
Command: service aws-kinesis-agent status

Step 4 – Switch user to ubuntu and create dummy log

su - ubuntu
cd /opt
mkdir applogs
cd applogs
vi myapp.log, press i to start inserting dummy lines, when done press Esc and use :wq! to save and exit

Step 5 – Create Data Stream

• Navigate to Amazon Kinesis, click on Create Data Stream
• Use demodatastream as the name
• Enter 1 for Shard value
• Click on Create data stream

Step 6 – Create Delivery Stream

• Click on Delivery Streams on the left menu
• Click on Create delivery stream
• Use demodeliverystream as the name
• Select source as Kinesis Data Stream and choose the data stream you created in Step 1
• Click Next, leave everything as is, Click Next
• Select S3 as the destination
• Choose an existing bucket or create new
• Click Next
• Choose Create or update IAM role
• Click Next Review and click on Create delivery stream

Step 7 – Configure Kinesis Agent to read dummy log

sudo vi /etc/aws-kinesis/agent.json
update kinesis.endpoint (https://docs.aws.amazon.com/general/latest/gr/ak.html)
update firehose.endpoint (https://docs.aws.amazon.com/general/latest/gr/fh.html)
update filePattern to /opt/applogs/myapp.log*
update kinesisStream to demodatastream
update filePattern to /opt/applogs/myapp.log*
update deliveryStream to demodeliverystream

Step 8 – Start Kinesis Agent and monitor Agent log

Attach Kinesis and Firehose full access and S3 full access roles to EC2 IAM Role

sudo service aws-kinesis-agent start
sudo tail -f /var/log/aws-kinesis-agent/aws-kinesis-agent.log

Step 9 – Open new SSH session to the same Ubuntu Instance

cd /opt/applogs
vi myapp.log, press i to start inserting dummy lines for kinesis agent, when done press Esc and use :wq! to save and exit

Switch window to see if Agent log shows parsed records

Step 10 – Verify Kinesis Agent is working

• Navigate to Amazon S3 dashboard
• Click on the bucket used as the destination for Firehose
• Dummy lines from myapp.log should show up here