Tardis scheduling policy - Revision history

import>Ghb24 at 17:02, 11 September 2008

2008-09-11T17:02:53Z

← Older revision		Revision as of 17:02, 11 September 2008
Line 75:		Line 75:

	--[[User:cen1001\|Catherine]] 16:50, 11 September 2008 (BST)		--[[User:cen1001\|Catherine]] 16:50, 11 September 2008 (BST)


			I like the look of the above suggestion - lumping everyone together under the portfolio group was a little harsh on people who wanted to use it occasionally, but were blocked out by others in the group.
			--[[User:ghb24\|ghb24]] 18:02, 11 September 2008 (BST)

import>Cen1001: /* Things we could change and their likely effects */

2008-09-11T15:50:05Z

Things we could change and their likely effects

← Older revision		Revision as of 15:50, 11 September 2008
Line 72:		Line 72:
	\|}		\|}

	We will also increase the reservation depth to 3.		We will also increase the reservation depth to 3. I will record the current usage and compare it with the usage in three months to see how it changes.

			--[[User:cen1001\|Catherine]] 16:50, 11 September 2008 (BST)

import>Cen1001: /* Things we could change and their likely effects */

2008-09-11T15:49:18Z

Things we could change and their likely effects

← Older revision		Revision as of 15:49, 11 September 2008
Line 54:		Line 54:

	Increasing the reservation depth to 3 sounds the easiest thing to do first. We can see how that works. If this reduces the overall usage too much we can reduce it to 2. --[[User:cja49\|Chris]] 12:10, 11 September 2008		Increasing the reservation depth to 3 sounds the easiest thing to do first. We can see how that works. If this reduces the overall usage too much we can reduce it to 2. --[[User:cja49\|Chris]] 12:10, 11 September 2008

			After some discussion with the people who put in the money for the machine we've agreed to alter the fairshare groups to reflect current reality rather the past. The portfolio group is to be abolished. A share of the machine worth 10K (or about 4%) will go into an 'other' group. The rest will be split between Ali and Michiel's groups. This leaves us with:

			{\| border=1
			\|-
			\|Group\|\|Fairshare %
			\|-
			\|Michiel\|\|30
			\|-
			\|Ali\|\|17
			\|-
			\|Jochen\|\|14
			\|-
			\|Stuart\|\|35
			\|-
			\|Other\|\|4
			\|}

			We will also increase the reservation depth to 3.

import>Cja49: /* Things we could change and their likely effects */

2008-09-11T11:11:04Z

Things we could change and their likely effects

← Older revision		Revision as of 11:11, 11 September 2008
Line 53:		Line 53:


	Increasing the reservation depth to 3 sounds the easiest thing to do first. We can see how that works. If this reduces the overall usage too much we can reduce it to 2. Chris		Increasing the reservation depth to 3 sounds the easiest thing to do first. We can see how that works. If this reduces the overall usage too much we can reduce it to 2. --[[User:cja49\|Chris]] 12:10, 11 September 2008

import>Cen1001: /* Things we could change and their likely effects */

2008-09-10T15:24:38Z

Things we could change and their likely effects

← Older revision		Revision as of 15:24, 10 September 2008
Line 44:		Line 44:
	* Increase the reservation depth to 2 or 3, causing the system to make reservations for the top 2 or 3 queued jobs, not just the top one. This will reduce overall utilization but on average help larger jobs because they won't keep losing their reservations. Small jobs tend to run fast enough that reservations are not an issue for them.		* Increase the reservation depth to 2 or 3, causing the system to make reservations for the top 2 or 3 queued jobs, not just the top one. This will reduce overall utilization but on average help larger jobs because they won't keep losing their reservations. Small jobs tend to run fast enough that reservations are not an issue for them.

	* Give a straight priority boost to large jobs to try to keep them at the top of the queue. More complex to configure. Would need to decide how much is enough and define a 'large job'.		* Give a straight priority boost to large jobs to try to keep them at the top of the queue. More complex to configure. Would need to decide how much is enough and define a 'large job'. Dilutes fairshare.

	* Increase the multiplier we give to 'time spent waiting' in the priority calculation so that if a job waits a very long time it can't be overtaken by jobs with better fairshare. This is simple and easy to understand. It will dilute the effect of fairshare to some extent. (added 4pmish 10/09/08, based on a suggestion from Nick Maclaren)		* Increase the multiplier we give to 'time spent waiting' in the priority calculation so that if a job waits a very long time it can't be overtaken by jobs with better fairshare. This is simple and easy to understand. It will dilute the effect of fairshare to some extent. (added 4pmish 10/09/08, based on a suggestion from Nick Maclaren)

import>Cen1001: /* Things we could change and their likely effects */

2008-09-10T15:23:57Z

Things we could change and their likely effects

← Older revision		Revision as of 15:23, 10 September 2008
Line 45:		Line 45:

	* Give a straight priority boost to large jobs to try to keep them at the top of the queue. More complex to configure. Would need to decide how much is enough and define a 'large job'.		* Give a straight priority boost to large jobs to try to keep them at the top of the queue. More complex to configure. Would need to decide how much is enough and define a 'large job'.

			* Increase the multiplier we give to 'time spent waiting' in the priority calculation so that if a job waits a very long time it can't be overtaken by jobs with better fairshare. This is simple and easy to understand. It will dilute the effect of fairshare to some extent. (added 4pmish 10/09/08, based on a suggestion from Nick Maclaren)

	Please add any comments below. I'd like to get any changes agreed by 24th September.		Please add any comments below. I'd like to get any changes agreed by 24th September.

import>Cen1001: /* Things we could change and their likely effects */

2008-09-10T13:04:15Z

Things we could change and their likely effects

← Older revision		Revision as of 13:04, 10 September 2008
Line 45:		Line 45:

	* Give a straight priority boost to large jobs to try to keep them at the top of the queue. More complex to configure. Would need to decide how much is enough and define a 'large job'.		* Give a straight priority boost to large jobs to try to keep them at the top of the queue. More complex to configure. Would need to decide how much is enough and define a 'large job'.

			Please add any comments below. I'd like to get any changes agreed by 24th September.

			--[[User:cen1001\|Catherine]] 14:04, 10 September 2008 (BST)

import>Cen1001: /* Node groups */

2008-09-10T12:52:37Z

Node groups

← Older revision		Revision as of 12:52, 10 September 2008
Line 30:		Line 30:
	=== Node groups ===		=== Node groups ===

	We force multinode jobs run on a set of nodes that are all attached to the same Infiniband switch. This can cause very large jobs (64 and 32-way) to take a long time to start because the system has to clear most of a switch for them. Other jobs may appear to jump the queue in this situation but if you check then those jobs will be being started on switches other than the one the system has picked for the top job. The system picks the optimum switch for the job, ie the one where the job can start soonest if all the other jobs in the system run to their maximum walltime. The calculation is redone each time the system state changes.		We force multinode jobs to run on a set of nodes that are all attached to the same Infiniband switch. This can cause very large jobs (64 and 32-way) to take a long time to start because the system has to clear most of a switch for them. Other jobs may appear to jump the queue in this situation but if you check then those jobs will be being started on switches other than the one the system has picked for the top job. The system picks the optimum switch for the top job, ie the one where the job can start soonest if all the other jobs in the system run to their maximum walltime. The calculation is redone each time the system state changes.

	== The current problem ==		== The current problem ==

import>Cen1001: /* Priority */

2008-09-10T12:51:07Z

Priority

← Older revision		Revision as of 12:51, 10 September 2008
Line 9:		Line 9:
	Priority is currently: 20 * ( personal fairshare result + group fairshare result ) + job expansion factor .		Priority is currently: 20 * ( personal fairshare result + group fairshare result ) + job expansion factor .

	Every individual user has a fairshare target of 20% of the machine. If you go over that usage then your fairshare result is negative; under it and it's positive. Everyone is also in one of four groups of users which have group fairshare targets. These are based on how the machine was funded: the 'stuart' group (Stuart Althorpe's research group) gets 38%, the 'michiel' group (Michiel Sprik) gets 20% (this includes their share of the Portfolio funding), the 'jochen' group (~~Jochem~~ Blumberger) gets 14% and the 'portfolio' group (everyone else) gets 28%. Your group and your personal result are added, so other people in your group can make your fairshare negative by running lots of jobs.		Every individual user has a fairshare target of 20% of the machine. If you go over that usage then your fairshare result is negative; under it and it's positive. Everyone is also in one of four groups of users which have group fairshare targets. These are based on how the machine was funded: the 'stuart' group (Stuart Althorpe's research group) gets 38%, the 'michiel' group (Michiel Sprik) gets 20% (this includes their share of the Portfolio funding), the 'jochen' group (Jochen Blumberger) gets 14% and the 'portfolio' group (everyone else) gets 28%. Your group and your personal result are added, so other people in your group can make your fairshare negative by running lots of jobs.

	Fairshare targets and usage can be seen by running 'diagnose -f'. The research group targets as labelled as ~~QAS~~ targets rather than group targets- this is a technical consequence of the way we set the system up. 'group' to Maui means 'Unix group' not 'research group'.		Fairshare targets and usage can be seen by running 'diagnose -f'. The research group targets as labelled as QOS targets rather than group targets- this is a technical consequence of the way we set the system up. 'group' to Maui means 'Unix group' not 'research group'.

	The fairshare calculation works on 48-hour slots (the longest queue is 48 hours) and uses the last ten. Older slots count for less than recent ones.		The fairshare calculation works on 48-hour slots (the longest queue is 48 hours) and uses the last ten. Older slots count for less than recent ones.

import>Cja49: /* Things we could change and their likely effects */

2008-09-10T10:47:08Z

Things we could change and their likely effects

← Older revision		Revision as of 10:47, 10 September 2008
Line 41:		Line 41:

	=== Things we could change and their likely effects ===		=== Things we could change and their likely effects ===


	* Increase the reservation depth to 2 or 3, causing the system to make reservations for the top 2 or 3 queued jobs, not just the top one. This will reduce overall utilization but on average help larger jobs because they won't keep losing their reservations. Small jobs tend to run fast enough that reservations are not an issue for them.		* Increase the reservation depth to 2 or 3, causing the system to make reservations for the top 2 or 3 queued jobs, not just the top one. This will reduce overall utilization but on average help larger jobs because they won't keep losing their reservations. Small jobs tend to run fast enough that reservations are not an issue for them.

	* Give a straight priority boost to large jobs to try to keep them at the top of the queue. More complex to configure. Would need to decide how much is enough and define a 'large job'.		* Give a straight priority boost to large jobs to try to keep them at the top of the queue. More complex to configure. Would need to decide how much is enough and define a 'large job'.


			Increasing the reservation depth to 3 sounds the easiest thing to do first. We can see how that works. If this reduces the overall usage too much we can reduce it to 2. Chris