How about you give us some more detail - tell us how different the hits and docs are. You're also probably better off querying http://127.0.0.1:9200/_stats/docs?pretty, as that gives you much more detail about the kinds of documents (primary, total, deleted). Does your Kibana index pattern match the index names correctly? Are you indexing daily/monthly/weekly/just one big index?

@MarkHenderson Thank you for your comment, I updated my question with your request.

You haven't told us what the differences is though. How many hits is Kibana showing you?

Kibana "hits" count is about 10% less.

You don't have date-based indexes, so I don't have enough information to tell you exactly, but my guess is that there is something being excluded by your Kibana range. Maybe you are filtering on a specific document type, or there is data outside of your date range that you're not expecting.

I use * as my filter/search query and same * as my index pattern, Kibana shouldn't exclude anything. How can I find what's being excluded by Kibana?

That's... not how Kibana is meant to be used. Do you actually have any time-based fields in your data?

Hello Mark

How are you?

Hi

I do have time based field inside of my data

I would suggest creating a few different index patterns: one, two and three. Look inside each index and see if Kibana gives you accurate hit counts there

Thank you for taking time with my question)

You caught me at a good time. I'm doing some maintenance on some of our Elastic indexes that have about a billion documents per index, so I have some time to spare

heh

I have 3 time based field and I tried with all 3 still not getting what I want

is there a way to search ES for items that do not have time based entry?

Create an index pattern that matches one of your indexes exactly (e.g. one) and see how you go with that

that should never be the case, but just to check

There is, but you can't mix time and not-time based documents in one index pattern

An index should only have time-based or non-time based documents in it, if you want to be able to browse it in Kibana. Otherwise Kibana will skip the documents that it can't determine the time for.

i tried that, that did not help me either

Or, you can turn time-based indexing off in Kibana

we have timestamps in each document and not just one, but 3)

Are they populated in every document?

Or could the missing docs be null?

they should never be any document that is missing timestamp

For the index one, you have 113227 undeleted documents. How many does Kibana say?

let me check

hmm, whenever I'm trying to add one index, I get data from another index now

You might have some mixup in the data inside the indices?

let me add * index and do simple search

possibly, im going have to look more into that

Otherwise, the browser plugin sense will allow you to query your data with more power (but less UI) than Kibana can

ahh wait

It might be useful to just see whats actually inside the index

no i did something bad

its ok now)

ok

I now got 141 hit

That's not very many

and expected number is 113227

Still got "last 5 years" selected?

yes

There is an Elastic plugin that allows you to actually browse the data differently

Just trying to find it

ok

github.com/appbaseio/dejaVu

I'd try and get that plugin installed and see if you can query your data through that UI instead.

It's more forgiving than Kibana, but doesn't do visualisations or that sort of stuff

That's all I can do I'm afraid. I'm 99% sure there's just something simple wrong in the documents, because I haven't seen kibana "miss" documents before

Hopefully you can find it by exploring the docs themselves

i got dejavu chrome app going