why are bucket files not processing?

Document ID : KB000117267
Last Modified Date : 17/10/2018
Show Technical Document Details
Introduction:
200 recent bucket files (48 hours is the oldest) in the reporting directory do not seem to be processing.
There are no corruption errors in the tomcat log and MYSQL.out shows no errors.
Some records are being processed but slowly and we are left with a lot of unprocessed buckets building up.
In busy environments, there is usually a build up of buckets, waiting on an insert to the reporting DB but the MySQL process list shows, that we are not waiting on a queue of inserts.

mysql> show full processlist; 
Looking at the bucket files in the reporting directory, all 230 are model availability buckets e.g. 

Line 343: -rw-rw----. 1 nmsdist users 8854 Sep 25 09:36 modelavailbucket1092616192_1537860988424.frm 
Line 344: -rw-rw----. 1 nmsdist users 5504160 Sep 25 09:56 modelavailbucket1092616192_1537860988424.MYD

The buckets are not showing in the bucket activity log and the log seems to be from updating the OS level.

-rw-r-----. 1 nmsdist users 8922 Sep 26 16:24 bucketactivitylog.frm 

It should also be checked at the application level, via MySQL command on the reporting DB. 

show tables 
 
Question:
Why are bucket files not processing?
Answer:
As the build-up of bucket files increases day continuously, we also see other buckets are being processed.
With no corruption errors in the tomcat log and nothing the MySQL log, the DB seems okay.
Looking further at the tomcat log we an error on one of the archive managers.

Sep 27, 2018 11:19:13.019 (SRM/EventPoller/xxxxx) (SRM_Events) - (ERROR) - Primary ArchiveManager seems to be down and connected to the Non-Primary ArchiveManager. and Event processing is not supported in this case. So, skipping this processing cycle. 

After resolving the Archive Manager problem, the bucket files were processed.

We can check how and when the bucket files were processed in the bucketactivity log. 
the example below uses a bucket file table on my lab so would need to be changed for one from the list of previously unprocessed buckets in the logs at mysqlreporting.out.

mysql> select * from bucketactivitylog where bucket_table_name like 'modelavailbucket16777216_1530705239861'; 
+--------+----------------------------------------+-------------+------------------+---------------------------+---------------------+---------------------+--------------+ 
| log_id | bucket_table_name | landscape_h | handler_name | finished_inserting_events | creation_time | destroy_time | event_log_id | 
+--------+----------------------------------------+-------------+------------------+---------------------------+---------------------+---------------------+--------------+ 
| 1253 | modelavailbucket16777216_1530705239861 | 16777216 | modelavailbucket | 1 | 2018-07-04 13:53:59 | 2018-07-04 13:54:00 | 140 | 
+--------+----------------------------------------+-------------+------------------+---------------------------+---------------------+---------------------+--------------+ 
1 row in set (0.04 sec) 



or we can use creation time based on the timestamps on the bucket files we have in this directory listing.
e.g. 

select * from bucketactivitylog where creation_time BETWEEN '2018-09-25 09:36:00' AND '2018-09-27 10:35:00'; 

It would be good to see what landscape they were on and destroy time for example. 
Was it the downed archive manager landscape? 
Additional Information:

Logs needed when investigating bucket files.


1. run the following on MySQL of the OC SRM server: 

from a shell, navigate to the $SPECROOT/MySQL/bin directory 
Launch the MySQL command prompt: 
./mysql --defaults-file=../my-spectrum.cnf -uroot -proot reporting 

Run the following command to redirect the MySQL output to a TEXT file: 
mysql> \T queries.txt 

Run the following MySQL queries: 
mysql> SELECT * from landscape\G 
mysql> select count(1) from bucketactivitylog where destroy_time is null; 
mysql> select count(1) from bucketactivitylog where destroy_time is null and bucket_table_name like "security%"; 
mysql> select count(1) from bucketactivitylog where destroy_time is null and bucket_table_name like "alarm%"; 
mysql> select count(1) from bucketactivitylog where destroy_time is null and bucket_table_name like "ncm%"; 
mysql> select count(1) from bucketactivitylog where destroy_time is null and bucket_table_name like "modelcreatedestroybucket%"; 

Run the following command to stop and close the TEXT file: 
mysql> \t 

Locate and upload  the $SPECROOT\mysql\bin\queries.txt file 

2. Generate and upload the getinfo logs on the OC SRM 

there is a utility called getSpectrumInfo.sh. You will find it in 
<$SPECROOT>\bin\support 
Could you run the following on the problematic server. 
a. copy to your $SPECROOT folder. 
b. open a bash shell 
To open a linux bash shell on windows, from the run command, type 
bash -login 
c. run "./getSpectrumInfo.sh lite" 

3. what´s the disk usage?