Accelerate the Synchronization with the Steem Blocks

in witness-category •  3 years ago  (edited)

A few of my tools require pre-process the blocks into a SQLite relational database. It takes time, especially currently, there are more than 43 Millions blocks on steem blockchain.

Starting from Block 1 to Sync with the latest are expected to take days... But there are tips to accelerate the process.

Do it on your Witness Server

The first important tip is use your own witness server to send the API to. If you are using an external server, it takes time as the HTTPS request are slow (network hops are the bottom-necks).

First, on your witness server, you have to turn on the following plugins:

plugin = witness block_api webserver condenser_api

Second, you would need to expose the 8091 RPC port. Then you can make API calls to the localhost (which is your witness server)

Calling the localhost

Your script should make calls to instead of

Direct Calls instead of using Steem Class

instead of using Steem(nodes=node) you should make calls directly via the HTTP request object

data={"jsonrpc":"2.0", "method":"condenser_api.get_block", "params":[number_block], "id":1},json=data)

This is a lot faster especially you are making lots of calls!

Commit to Database every 1000 blocks

Instead of updating the database every block, you should only do this every 1000 blocks (of course this is an example number) or more.

You don't want the I/O (especially write) to slow down crawling the blocks.

Add Index After

Another tip is to add index after the data is sync-ed. It makes a huge difference especially if you have a complex configuration of the database indices.

I hope this helps!

Steem On!~

If you like my work, please consider voting for me, thanks! type in justyy and click VOTE

Alternatively, you could proxy to me if you are too lazy to vote!

Also: you can vote me at the tool I made:

Visit me at:

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

Great tip! I do the same for the SteemWorld backend and I also use SQLite ;) I've added a constant USE_LOCAL_NODE = true in my configuration class to be able to easily switch between my own and a public external one, in case my node is in maintenance.

@steemchiller @justyy
Is we have any voting bot for steem? I am looking for voting service for steem (curation purpose) because old services has discontinued like steemauto.

I was asked this just a few hours ago :)

  ·  3 years ago 


  ·  3 years ago 

Thank you. that would be great!

Would be nice seeing a tutorial on how to setup using the latest fork for steem blockchain. So we can encourage other members to build their own steem witness servers. Better approach for the stability of the chain.

Anyone done it. I haven't updated my tutorial to the latest fork.

  ·  3 years ago 

Yes, you are right.

Are you looking for an Expert to design you a fashion beauty website with full plugins, to know more kindly contact me on
#webdesigner , #webdesignagency , #webdeveloper , #webdesigning , #webdesign , #webdevelopment , #webdesigncompany , #webdesigntips , #webdesignservices , #webdevelopement