I saw an interesting thread today on an internal list that I would like to share with the world. After all, while each of us is a precious snowflake, our problems and challenges are not always unique. 
If one person has this question, there’s a good chance that someone else does as well!
Let’s the set the stage. You have a typical Splunk environment with a bunch of servers with forwarders installed on each one which grabs data and sends it to the indexers. Automation is important, so you want to ensure that the configuration on these forwarders is consistent. One way to do this is to use our Deployment Server feature, which uses a pull model to ensure that all forwarders check in periodically to a DS to receive updated config files. But how do you indicate which managed nodes (any system running a forwarder) is to receive the correct configuration according to their roles? The answer is by using server class filters, such as whitelists and blacklists.
So far, this is basic Splunk admin and has nothing to do with REST APIs. Splunk is nothing without the large systems around it which create data. A very important use of our REST APIs (and our higher level SDKs for a half-dozen different computer languages) is to integrate with these other systems and tools in your environment. Especially as your environment grows, you will want to automate every Splunk administration task that you can, and that includes updating server classes, of course. This can definitely be done using the REST API.
Now the reason I wrote this post was because RESTful APIs may not behave like you expect if you are used to writing scripts against a system with a non-RESTful API. Here are the two questions which were asked on that internal list today:
- Is there a way to append a whitelist entry to an existing server class? It appears that the endpoint (deployment/server/serverclasses/{name}) is restricted only to replacing all entries.
- How does Splunk handle two such requests occurring at the same time? Perhaps an error message be produced? Is there a file lock that we need to be aware of?
If you don’t know much about a RESTful API, these seem like perfectly logical questions. However, once you learn more about what REST actually means, you’ll see that you don’t even to ask these questions! I’ll explain. First, here’s what Wikipedia says:
In computing, Representational State Transfer (REST) is the software architectural style of the World Wide Web. REST gives a coordinated set of constraints to the design of components in a distributed hypermedia system that can lead to a higher performing and more maintainable architecture.
Am I right? Get it?
I’m kidding, I don’t even know why I pasted that! Perhaps I’m a bit of a sadist. Skipping ahead, here’s what you, the Splunk administrator, need to know about REST in terms of automating Splunk, and it should explain why the above questions don’t apply.
The answer to both of these questions is pretty much embedded into the definition of a RESTful API. The interface (the “I” in API) is built around the HTTP protocol. That protocol has a set of verbs: POST, GET, PUT, DELETE aka methods, which most folks already understand intuitively. GET gets, and so on. APIs written using a RESTful model tend to adhere to a pattern where PUT is used to update an existing resource. And the way that you typically do updates is to send the entire newly updated record as part of the PUT message. Why? Because of the “S” in REST which stands for Stateless. Because you can’t assume the past state as you can with a purely transactional system, you have to do it this way and include all of the data needed for an update. (You can read more about REST statelessness here.)
Therefore:
- No appending in place to an item, that’s not generally done in REST.
- No locking! Last update wins, always. Locking only makes sense when you are performing partial updates.
Taking the theory home to practical Splunkiness, does this mean that the Splunk REST API behaves this way? It really should, because that’s not only what external developers expect, it’s also what the various different teams within Splunk Engineering expect of each other’s REST interfaces. If these assumptions were false, that would be a bug, and perhaps a serious one.
If you want to read more about working with Splunk REST APIs, check out our REST API tutorials!
-hal, dev evangelist for Splunk