AIX includes a structure for elaborate limits to be setup for process resources. These limits are defined in the file /etc/security/limits (which may be modified directly or via the command chuser).
example of output:
fsize = 2097151
core = 2048
cpu = 3600
data = 131072
rss = 65536
stack = 8192
These settings specify absolute limits on resource usage, and they cannot be overridden by the user.
To change your user's memory use limit, you can use the following command:
# chuser rss=102400 username
This command sets username's default memory use limit to 100MB by modifying or adding the rss line for their username in /etc/security/limits. As usual, the limits set in the default stanza are applied for any user without specific settings of their own. Setting a limit of -1 will allow unlimited use of that system resource.