Will AI replace us? Using ChatGPT to create Python Actions for unSkript

Over the last few weeks, ChatGPT has been the buzz of the development world.  One of the biggest questions that has been asked is “Is the AI coming to replace developers?”  Will a fleet of Daleks come after us shouting “exterminate?” Or will AI code generators just become another tool in our arsenal? In this post, we’ll look at the code generated by ChatGPT, and how good (or not good) it actually is.

If you have not yet tried ChatGPT, you can sign up at OpenAI for a free account, and then head to https://chat.openai.com/chat and start asking questions.  The AI answers in a conversational way, and remembers previous answers, allowing you to  build on top of previous answers and refine the result. If you plan to use ChatGPT to generate code, expect to use this feature a lot.

If you ask Chat GPT to create code for you, it will, and it will even add comments, and describe what the code is doing.  At unSkript, we are building a CloudOps system with hundreds of Actions that can be put together in unique ways – connecting your services with observability and automated remediation. As a result, we are releasing dozens of new Actions (microservices that complete a distinct function) into our Open Source Repository. 

Since unSkript’s Actions are written in Python (and not a domain specific language), I thought it would be neat to see if ChatGPT might be able to help us accelerate the creation of these Actions.  

Warming up

Like any new tool, you have to understand what makes ChatGPT ‘ticks’ in order to make it work for you. Let’s begin by trying to get GPT to copy some Actions that we have recently blogged about – analyzing and adding tags to AWS EC2 resources.

Lets start with asking to check an EC2 instance for a specific tag:

ask chatgpt for a scrip[t to get tagged ec2 instances

The result is nearly exactly the same code as what was used in our blog post (and our published Action):

chatGPT response

However, the code only works for one EC2 instance, and our Action needs to look at all of the EC2 instances in our account. So we can ask for it to be extended, and chatGPT comes up with an acceptable response to get every EC2 instance, and see if there is a “CostCenter” Tag.

chatGPT query

Python output from chat GPT

I find the variable naming system here a bit counterintuitive – naming the list of reservations ‘instances’ and then naming each instances ‘reservation‘, but otherwise this is lock step with the code that humans used to iterate through all of our EC2 instances and then determine the presence or absence of a tag.

Tagging the Instances

We could take the results from this Action, and then build another Action to add the CostCenter tag to those instances that do not have the tag.  Let’s ask the bot to help:More chatGPT screenshots

The bot has nailed it again! This can be copied right into the unSkript interface, and it’ll work. Now, let’s ask GPT to abstract this a bit, and make the key:value pair variables, rather than hard coded.  This again works like a charm:

make the tags variables. This is extending the code from the last question

Pushing our GPT queries into new Directions

By using Chat GPT to duplicate Actions that already exist, we are able to better understand how to generate results that might work when we branch into new Actions that have not yet been written.  Let’s keep our queries to AWS and EC2 CloudOps, but build an Action to find out the age of all of our EC2 instances.

get age of all ec2 instances

The Chatbot’s code is similar to the code it used in the earlier example (with the same funny variable naming schema).  And, like a good computer,  it does what I asked it to do -> it will print a line for each instance telling me the age. This is great, but not terribly useful for further usage. Perhaps you’d like to restart EC2 instances that have been running for >180 days (the old “turn it off and turn it back on” approach), so having this list in a way that can be manipulated would be useful.  So I defined a JSON format that I would like: key is instanceID, and the value is the age.  I then asked Chat GPT to return the results in that way:

change result from lines of text to a dict

Copying this code into unSkript, this works a charm.

Monitoring CPU usage

Another issue that can arise with EC2 instances is that the CPU is too high.  Can ChatGPT build an Action to list the CPU usage of each EC2 instance?  This will use two calls to AWS – one call to EC2 to get the list of instances, and then a 2nd round of calls to CloudWatch to get the CPU utilization.

The first time I asked Chat GPT to generate this code – it did so flawlessly.  But, when getting screenshots for this post, it failed.  It attempted to make the 2nd set of calls to the EC2 endpoint:

incorrect code from chatbot

In the 2 red boxes, both calls are being made to the EWC2 endpoint. This is easily corrected with a little prodding. Now there are 2 calls, one to EC2 and the other to cloudwatch:

corrected the chatbots code

Once corrected, this runs.  However, not every EC2 instance responds with a CPU usage, so a short error check (is the array’s length >0) to populate the data prevents the Action from breaking in practice:

 

               # Calculate the average CPU usage for the past 24 hours

                #error check for the presence of CPU  usage data

                if len(response['Datapoints'])>0:               

                    cpu_utilization_values = [datapoint['Average'] for datapoint in response['Datapoints']]

                    avg_cpu_utilization = sum(cpu_utilization_values) / len(cpu_utilization_values)

                    results[instance_id] = avg_cpu_utilization

                else:

                    results[instance_id] = "error"

The response from AWS has 24 entries, and here we have averaged them all to get the full 24 hour average.

Trust, but verify

Note: in some iterations, the bot did not do the average calculation and just assigned [datapoint[‘Average’][0] as the average (the first entry in the array – not the true average).  So, it is very important to check all the inputs and outputs used the the bot, and ensure that it is using them correctly. 

One last example: Bot goes a bit off the rails

This last example should have been quite similar to the previous one – checking the Network data in and out of each EC2 instance over the last hour:

chatbot query

I expected the code to first query EC2 for the list of instances, and then hit up Cloudwatch for the data.  But we got a bit of a mashup between a SQL query and Python in the result.  Instead of a for loop over each instance, we got:

is it python? Is it SQL?
Is it a SQL query?  Or is it Python?  Needless to say, this does not work.

However, the Cloudwatch query to get 2 the metrics (bytes in and out of each instance) does work, and the code correctly outputs the data as dict.  By reusing the for loop that the BOT created in previous queries, and switching in the new query, I was able to get the code to run.  However, in error checking, I found that sometimes CloudWatch returns a null value tor a result. Adding a little error correction at the end fixes this. Also - CloudWatch reports the values in Bytes, so we convert to MB for easier comparisons.

  if len(metrics['MetricDataResults'][0]['Values'])>0:

                    NetworkInMB = round(float(metrics['MetricDataResults'][0]['Values'][0])/1024/1024,2)

                else:

                    NetworkInMB = "error"

                if len(metrics['MetricDataResults'][1]['Values'])>0:    

                    NetworkOutMB = round(float(metrics['MetricDataResults'][1]['Values'][0])/1024/1024,2)

                else:

                    NetworkOutMB = "error"

                metricsIwant = {metrics['MetricDataResults'][0]['Label'] : NetworkInMB, metrics['MetricDataResults'][1]['Label'] : NetworkOutMB}

                result[instance_id] = metricsIwant




Conclusion

Are the bots coming for our jobs?  In these few examples, ChatBot generally did ok.  There were certainly bugs in the code that was generated, and a solid code review of what the bot has generated must be undertaken to ensure that it is providing what was asked for.

That said, generating 90% of the code correctly allows us to spend that extra time error checking and verifying (that we should be doing anyway).  

Using ChatGPT allowed me to create these Python unSkript Actions in a fraction of the time that it would have taken me to do them solo.  By generating much of the boilerplate code, I could take the extra time to ensure that the results made sense.

Want to try to build your own CloudOps Actions with unSkript?  Sign up for our free trial.  And look for the Actions built in this post using ChatGPT to appear in our Awesome-CloudOps-Automation Github repository in the very near future!

Share your thoughts