It is not possible to set permissions for charts
However, by setting permissions for records and fields, you can make it so records are not displayed in aggregated data if a user does not have permission to view them.

Only the records and fields that a user has permission to view are subject to data aggregation.
If a field that a user does not have permission to view is specified in data aggregation settings, the field will be excluded from the data aggregation process and will not be displayed in the aggregated data.