Our website freelensia.com is still using Harmony for calendar management.
Last weekend we faced a mass-signup spam attack (we did not have GG capcha for signup), which resulted in 60K new users added to the database. We tried to delete the fake users using a rake task but it did not work so well.
We manually scanned the dumped .sql file and confirmed that the fake users only showed up in:
- people
- community_memberships
- active_sessions
- emails
- user_custom_profile_values
So we used a DELETE method to delete them from each table using Heidi SQL. After a server restart, we had to fix Sidekiq which was still sending emails to those fake users, but all went well.
But Harmony does not work anymore. When saving a listing with a calendar, it says “Failed to Connect to the book Service”. Puma Error log shows this: https://pastebin.com/E6KyhGA1
We tried to restart it from our Monit dashboard. The process shows RUNNING but on the website calendar is not available.
We tried a hard restart with:
sudo su
cd /home/ubuntu/app/harmony
sudo docker-compose up
We got this log: https://pastebin.com/DQ1TkXQN
We had a bunch of errors like:
api_1 | at clojure.main.main(main.java:37)
api_1 | Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Unknown database 'harmony_db'
or
api_1 | 06:18:48.983 [main] INFO com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Closed.
api_1 | 06:18:48.991 [main] ERROR harmony.errors - {"component":"error-reporter","event":"uncaught-exception","data":{"msg":"Failed to encode log event data as JSON!"}}
api_1 | clojure.lang.ExceptionInfo: Error in component :db-conn-pool in system com.stuartsierra.component.SystemMap calling #'com.stuartsierra.component/start
api_1 | at clojure.core$ex_info.invokeStatic(core.clj:4617)
Just FYI we ran out of space on our server, so we emptied the content of some log files such as:
truncate -s 0 /home/ubuntu/app/freelensia/shared/log/sidekiq.log
truncate -s 0 /home/ubuntu/app/freelensia/shared/log/sidekiq.log.1
sudo su
truncate -s 0 /var/lib/docker/containers/c7a8c30d1c26caafc07b6204d2ad5e615a5cf5778bed5d66b784091d5465e5e3/c7a8c30d1c26caafc07b6204d2ad5e615a5cf5778bed5d66b784091d5465e5e3-json.log
These files had 16GB, 1.4GB and 6GB sizes, respectively. These files had 16GB, 1.4GB and 6GB sizes, respectively. We also downsized our EBS by swapping with a smaller EBS volume following this guide. But the EC2 instance and the website works fine after that, so that is probably not the problem.
Anybody with ideas?
