Start a new topic

Rotation and time constraints

Backstory:

2 servers, each with 900+ databases ranging from about 10GB to a few TB.

Not much traffic at night due to the nature of our business (heavies usage would be between 7 AM East coast time, to about 7 PM West coast.  No idea if we have users in Hawaii, Alaska or territories, if we do not many).


1 server with fewer databases, but one that is 7 TB.  Regular CHECKDB gets through everything but this one database.


My main concern right now is the 2 servers with tons of databases.


Here is what I am thinking, with fine tuning as I see it in action:

Dynamic thresholds around 100 GB (default) for switching to TABLECHECK.  This should allow most of the smaller client databases to run CHECKDB.


Rotational Scheduling, where it run 3 hours (1 AM to 4 AM) per day during weekdays, 8 hours (1 AM to 9 AM) during the weekend.  Weekend some time would be interrupted earlier if there is patching on the server, being able for it to restart if it does would be optimal.


This is what I have done on my own test server to test it, hopefully not too far off.

 

update Minion.CheckDBSettingsRotation 
	set IsActive = 1, -- activate Rotation
	RotationMetricValue = 180
where RotationLimiter = 'Time'

  

update Minion.CheckDBSettingsServer
	set IsActive = 0
where OpName = 'CHECKDB' 
	and DBType = 'User'

 

 

update Minion.CheckDBSettingsServer
	set IsActive = 1, -- to activate
	BeginTime = '01:00:00', 
	EndTime = '04:00:00',
	Day = 'Weekday' -- was saturday
where OpName = 'AUTO' and ID = 3

 

  

insert into Minion.CheckDBSettingsServer (
	DBType, 
	OpName, 
	Day, 
	ReadOnly, 
	BeginTime, 
	EndTime, 
	MaxForTimeframe, 
	CurrentNumOps, 
	NumConcurrentOps, 
	DBInternalThreads, 
	Debug, 
	FailJobOnError, 
	FailJobOnWarning, 
	IsActive)
select
	DBType, 
	OpName, 
	'Weekend', 
	ReadOnly, 
	'01:00:00', 
	'09:00:00', 
	MaxForTimeframe, 
	CurrentNumOps, 
	NumConcurrentOps, 
	DBInternalThreads, 
	Debug, 
	FailJobOnError, 
	FailJobOnWarning, 
	IsActive
from Minion.CheckDBSettingsServer S
where 
	S.OpName = 'auto' -- yeah redundant
	and S.ID = 3 

 

 So I set the Rotation for only Time to be IsActive = 1.


Leave the System CHECKDB active from 22:00 to 22:30.


AUTO active for the User databases, two records, one for Weekday and one for Weekend.


On these 2 servers, I THINK I could increase NumConcurrentOps and DBInternalThreads, only backups should really be running at that time of night, with maybe a few occasional users working way too late.

Comments and suggestions welcome.  First time I am implementing one of your solutions.



So the missing piece of your scenario is being able to change the rotation runtime on the weekend.  To do that you'll want to create an SP that makes the change in the Rotation table.  Then, call that SP in the BatchPrecode col of the SettingsServer table for that weekend row.  

You'll have 2 rows for checktable in the Rotation table and you'll toggle which one is active.

Your SP will check the day, and if it's a weekend day, it'll set that row active, and the weekday row inactive, and vice versa.


Now, you can handle the other toggle a couple ways.  You can either have a PostBatch code that changes it back when it's done, or you can put the precode in the weekday row as well and it'll always check for the day.


Does that make sense?


1 person likes this

Yupp!  Makes perfect sense.


Added a proc called Minion.SetActiveRotation 

 

update Minion.CheckDBSettingsServer 
set BatchPreCode = 'exec Minion.SetActiveRotation' -- called with defaults.  Can be used to also update the time limit
where IsActive = 1

 




Nice, however I'd put those comments both in the SP itself, and in the comment col of the SettingsServer table. And if you need it to do anything else just add it to that SP. Let me know if it does what you expected.

Yeah they are in the comments of the proc, the comment would just be in my file where I run the query, not in the BatchPreCode field.

Ran last night on my test production server (don't ask) last night.  Mostly ran as expected.   Did TABLECHECK on the database I expected it, but in CheckDBLog, for the user databases does shows "None" for the RotationLimiter.  For the System databases it shows "Time".


From the CheckDBSettingsRotation table:

image

Does there need to be an "AUTO" record in there as well with limits?  The System DBType is set to run CHECKDB, while the User is set to AUTO.


From CheckDBSettingsServer:

image



Ok, that might just be a logging bug.  Let me see if I can repro and I'll get back when I know something.


1 person likes this

Thanks, unfortunatly I don't have enough data on this server to see if it hit the time limit or not since it finished in 93 minutes vs. the 180 for the config.  I'll lower the limit to 60 for tonight and see what happens too.

Y but it still should've logged it correctly. I'm gonna do a test now and see.

Ok, I was able to repro.  Assuming you're on MC 1.1, give this fix a try.

sql

Thanks.  Will let it run in my test over the weekend.

Was it just a logging issue?  After I turned the time down to 30 minutes, the two tables that would take more time than allowed by config did not get run.

hmmm  Time limit is not being applied.  And it seems it did not run a rotation, i.e. the databases that got run Saturday morning got run again Sunday morning.


Tell me what you need.  Could supply you with a backup of the MinionWare database as it is the only thing in there.

From CheckDBLog:

image

And not every DB got run, yet on the run from this morning, it started to do the same ones over again.


Y send me a copy of that DB in email.  Don't post it here as you'll have DBNames in there and you don't want those public.  Unless you don't care.

Emailed the backup.

Login or Signup to post a comment