Sync PostgreSQL to ElasticSearch and CDC (change data capture)

--

Using Logstash ( this best option ), Synchronization postgreSQL to elastic search without duplicate ID, In the case that we are using our database as an input source for Elasticsearch, we may be interested in keeping our existing documents in-sync with our data as the database undergoes updates. In this case, we can simply index our rows in Elasticsearch with unique ids such that any time we re-index them, they will just update. This way, we prevent Elasticsearch from assigning a new ID for each record and generating duplicates!

Ref: https://www.elastic.co/blog/logstash-jdbc-input-plugin

Using Debezium to allow stream data

git: https://github.com/debezium/debezium

example: https://github.com/YegorZaremba/sync-postgresql-with-elasticsearch-example

Using PGSYNC

git: https://github.com/toluaina/pgsync

Sign up to discover human stories that deepen your understanding of the world.

Free

Distraction-free reading. No ads.

Organize your knowledge with lists and highlights.

Tell your story. Find your audience.

Membership

Read member-only stories

Support writers you read most

Earn money for your writing

Listen to audio narrations

Read offline with the Medium app

--

--

No responses yet