deleting a document from Elasticsearch data stream

courtesy from https://unsplash.com/@helloimnik

It sounds super simple to remove a document from an Elasticsearch data stream, isn’t it? But this is what happened this morning when I am trying to do so…

Me: Let’s run a delete like this~

DELETE log-basic-dev/_doc/7vIUq3wBugmWpr_4Z3ps

Kibana: bomber~ (lalalalala)

{
“error” : {
“root_cause” : [
{
“type” : “illegal_argument_exception”,
“reason” : “only write ops with an op_type of create are allowed in data streams
}
],
“type” : “illegal_argument_exception”,
“reason” : “only write ops with an op_type of create are allowed in data streams”
},
“status” : 400
}

datastreams supports “create” only?

Not exactly, datastreams by default assumes creation of data. Also it has strict rules on how to “create” the data as well:

  • only POST creation is accepted (i.e. no PUT operations)
  • if using _bulk api, the only op_type accepted is “create” (i.e. no “index”, “update” and “delete”)

So does it mean we can’t delete stale documents? If really have to, we would need to query out which backing index containing that document first. Usually a search query could identify which backing index it belongs to:

{
“took” : 5,
“timed_out” : false,
“_shards” : {… },
“hits” : {
“hits” : [
{
_index” : “shrink-wmox-.ds-log-basic-dev-2021.10.23–000007”,
“_type” : “_doc”,
_id” : “URVyq3wBdZHgTDXtPN0t”,
…}
]}}

by examining the meta data from the search response, we could get back the backing index’s name and its document id.

Me: Alright~ Now let’s delete that document directly from the backing index~

DELETE shrink-wmox-.ds-log-basic-dev-2021.10.23–000007/_doc/URVyq3wBdZHgTDXtPN0t

kibana: bleh~

{
“error” : {
“root_cause” : [
{
“type” : “cluster_block_exception”,
“reason” : “index [shrink-wmox-.ds-log-basic-dev-2021.10.23–000007] blocked by: [FORBIDDEN/8/index write (api)];”
}
],
“type” : “cluster_block_exception”,
“reason” : “index [shrink-wmox-.ds-log-basic-dev-2021.10.23–000007] blocked by: [FORBIDDEN/8/index write (api)];”
},
“status” : 403
}

Now what? Forbidden/8/index write (api). Talk to me (humans here)~

The error message is really… After some diggings, I found that the targeted index has been made read-only. As “shrunk” index usually is read-only for performance reasons. Hence, all I need to do now is to revert this behaviour.

PUT shrink-wmox-.ds-log-basic-dev-2021.10.23–000007/_settings
{
index.blocks.write”: false
}

and delete again…

DELETE shrink-wmox-.ds-log-basic-dev-2021.10.23–000007/_doc/URVyq3wBdZHgTDXtPN0t

kibana: congratulations~

{
“_index” : “shrink-wmox-.ds-log-basic-dev-2021.10.23–000007”,
“_type” : “_doc”,
“_id” : “URVyq3wBdZHgTDXtPN0t”,
“_version” : 2,
“result” : “deleted”,
“_shards” : {…},
“_seq_no” : 1,
“_primary_term” : 2
}

finally~

wrap up

In this scenario, I have revert the read-only behaviour of a shrunk index and therefore able to delete given documents within a datastream. However, for logical and performance reasons, once the deletion is done, the backing index should be re-applied to read-only (set index.blocks.write to true again)

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
devops terminal

devops terminal

114 Followers

a java / golang / flutter developer, a big data scientist, a father :)